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(54) Combination of 2-D detector array with display for image processing 



(57) An image display/processing apparatus (A) 
which integrates as a single unit a two-dimensional 
(2-D) display (B), such as an LCD, with a two-dimen- 
sional (2-D) sensor array (C), such as a photodiode ar- 
ray. The arrays are aligned with a pixel (38) correspond- 
ing to a single sensor element (34) of the sensor array 



(C). Levels from the sensor elements (34) which have 
been thresholded to predetermined threshold values 
are fed back (52; Fig.1 (d)) to the LCD (B) for image dis- 
play and processing. The LCD image (Fig. 2) may be 
moved in a pixel increment by control circuitry to imple- 
ment morphological image processing. 
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Description 

The present invention relates to the art of image dis- 
play and processing, and finds particular application in 
an image display/processing device for displaying and 
morphological processing of sensed images. 

Various types of image displays have been devel- 
oped, including liquid-crystal displays (LCDs) which are 
electronically switched displays making use of changes 
in the optical properties of liquid crystals in series with 
an electric field. A thin film of liquid crystals is sand- 
wiched between glass plates imprinted with transparent 
electrodes. When a voltage is applied selectively across 
the electrodes, the liquid crystal molecules between 
them are rearranged or switched in polarization so that 
light is either reflected or absorbed in the region bound- 
ed by the electrodes to form characters or graphics. Liq- 
uid crystals are chemical mixtures that behave like crys- 
tals in an electric field. 

Glass covers of LCDs are assembled with a prede- 
termined space between them to keep a constant dis- 
tance. They are then sealed around the edges to contain 
the viscous liquid crystals, and are equipped with polar- 
izing filters to assist in controlling image output. 

Most LCDs today employ twisted-nematic field-ef- 
fect (TNFE) liquid crystals that provide either dark or 
black characters on a lighted field. When subjected to 
an electric field, the TNFE molecules twist changing the 
optical properties of the light thereby varying reflection 
or transmittance of the light through the LCD. When the 
voltage is removed, they recover their normal orienta- 
tion and are transparent to light. The recovery time is 
called response time. 

In large LCD matrix displays, each pixel is common- 
ly a twisted-nematic cell. Each cell is formed as an x-y 
coordinate matrix of fine width parallel electrodes 
formed on the crystal sandwich of the opposing glass 
plates in a liquid crystal sandwich. These electrodes are 
formed by the deposition of a transparent conductive 
film on the inside of each plate, and subsequent photo- 
lithographic and etching processes. 

The glass plates are assembled so that addressa- 
ble parallel electrodes are at right angles to each other 
over the liquid crystal material. A pixel is switched when 
the voltage pulses across the opposing electrodes and 
liquid crystal exceed a predetermined voltage threshold. 
The LCD is organized so that the rows are pulsed cycli- 
cally and the data is multiplexed into the columns syn- 
chronously. 

Large LCD displays based on metal-insulator-metal 
(MIM) technology are known. Another active matrix 
technology is thinfilm t ran sistor(TFT) -based displays. 
These large-panel liquid-crystal displays use a transis- 
tor and capacitor at each pixel location to improve con- 
trast and readability. Conventional LCD pixels are driven 
only by a short voltage pulse during the scanning cycle 
but active-matrix pixels are driven by a continuous volt- 
age. The transistor and capacitor hold the pulsed-volt- 



age level indefinitely. 

As with image display technology, various types of 
image processing devices have been developed. One 
aspect of image processing is the need to electronically 
5 acquire the image which is to be processed. Arrays of 
sensing elements such as photodiode and photocon- 
ductors have been found to be suited for such image 
acquisition. 

It is also known that amorphous-silicon and polysil- 
10 icon based devices have been developed for a wide va- 
riety of applications, including image sensors and flat- 
panel displays. These devices take advantage of certain 
amorphous-silicon or poly-silicon film properties, such 
as low-temperature processing capability, high photo- 
75 conductivity, high field-effect switching ratio and dopa- 
bility. Moreover, due to the advent of large-area deposi- 
tion and photolithographic patterning systems, it is com- 
mon practice to use large substrates to fabricate these 
devices to improve productivity and gain cost savings. 
20 This large-area device technology has spawned 
several page-wide linear-array device applications in 
the fields of facsimile communication, document scan- 
ning, and page printing. Page-wide linear sensors using 
a Si:H have been used for some time in facsimile appli- 
es cations. For example, a-Si:H arrays of photodiodes in- 
dividually wirebonded to a number of CMOS LSI chips 
mounted on the same substrate have been developed. 

Commonly, sensor arrays employing photodiodes 
have the sensors positioned in rows and columns of a 
30 x-y coordinate configuration in a manner similar to that 
of LCD arrangements. Such sensor arrays are also driv- 
en in a manner corresponding to that of LCD displays. 

There are similarities between 2-D image displays, 
such as LCDs, and 2-D image sensors, such as photo- 
ns diode arrays when, for example, large-area amorphous- 
or poly-silicon electronics are employed. It has been 
found that the combination of these two technologies al- 
lows for very powerful image capture, display, and 
processing. Therefore, the present invention provides 
40 for the combination of these technologies through the 
integration of a 2-D sensor array with a 2-D image dis- 
play, whereby with appropriate combination a system is 
constructed for use as an ordinary flat panel display, and 
which also is capable of parallel image processing such 
45 as morphological processing. 

The present invention provides an image dis- 
play/processing apparatus comprising: a liquid crystal 
display (LCD) having an array of switching elements, 
corresponding to a plurality of LCD pixels; a sensor ar- 
50 ray integrated with the LCD to form a single unit, the 
sensor array having a plurality of sensor elements po- 
sitioned in rows and columns in the array, the array of 
switching elements and the array of sensor elements 
aligned with each other to allow a given LCD pixel to 
55 project onto a corresponding sensor element of the ar- 
ray of sensor elements; and driving circuitry in operative 
connection with both the LCD and sensor array. In one 
embodiment, the apparatus is configured to allow mor- 
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phological image processing operations to be imple- 
mented on images displayed on the LCD. 

In another embodiment the integrated device is 
used to provide tor document scanning by the 2-D sen- 
sor array on a side opposite the LCD. 

The invention further provides a method for display- 
ing and processing an image on an integrated dis- 
play/processing apparatus comprising the steps: load- 
ing a liquid crystal device (LCD) with image data, the 
LCD being in the form of an array with switching ele- 
ments positioned at rows and columns and with a pixel 
associated with each of the switching elements; display- 
ing an image by adjusting each of the pixels in accord- 
ance with the loaded image data; sensing data of the 
displayed image by a sensor array comprised of a plu- 
rality of sensing elements located at rows and columns, 
the sensor array integrated as a single unit with the LCD, 
the sensor array and LCD aligned with each other to al- 
low a given LCD pixel to project fully onto one sensing 
element of the sensing array; feeding back data from 
the sensor array to the LCD based on the data of the 
displayed image; and processing the feedback data of 
the displayed image. 

The method preferably further includes the step of 
driving the LCD and sensor array by a single driving cir- 
cuit, and more preferably in synchronism. Preferably, 
the driving step further includes shifting the image on 
the LCD, and more preferably further includes reading 
out a line of the sensor array at a same time as a corre- 
sponding LCD line is updated. 

The method preferably further includes dynamically 
programming the sensor array to integrated data ob- 
tained by the sensing elements for predetermined 
length of time corresponding, preferably, to a specified 
number of LCD frames. 

The method preferably further includes morpholog- 
ically processing the displayed image. The morpholog- 
ical processing includes: defining a structuring element; 
shifting the image as prescribed by the structuring ele- 
ment to generate multiple shifted images of the original; 
and performing boolen operations using the original im- 
age and the multiple shifted images. Alternatively or ad- 
ditionally, the morphological processing includes: inte- 
grating data obtained by the sensing elements over all 
translations of an original LCD image. Alternatively or 
additionally, the morphological processing for dilation in- 
cludes: zeroing the sensing elements; sequentially load- 
ing LCD images at a rate of 1/AT, where T equals time; 
integrating the loaded LCD images; reading the sensor 
array after time (N + 1 )AT; and thresholding the sensor 
array to a value of (1 ) when a sensor level corresponds 
to a pixel being in an ON state of the LCD for a period 
AT or greater, with other signals being thresholded to 
(0). Alternatively or additionally, the morphological 
processing, for erosion, includes: displaying on the 
LCD, at a rate of 1/AT, a set of N images translated by 
a predetermined inverted shift; reading out the sensor 
array after a time (N + 1 )AT; and thresholding the sensor 
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array to a value of (1 ) when a sensor level corresponds 
to a pixel being in the ON state of the LCD for a period 
(N + 1)AT, with smaller signals thresholded to (0). 

The invention further provides an image dis- 
5 play/processing apparatus when suitably programmed 
for carrying out the method of claim 10, or according to 
any of the particular embodiments described herein. 

A further understanding of the nature and advan- 
tages of the invention herein may be realized by refer- 
red ence to the remaining portions of the specification and 
the attached drawings, in which: 

FIGURE 1(a) is a top view of an image dis- 
play/processing apparatus according to the present 
is invention having a LCD and sensor array integrated 
as a single unit; 

FIGURE 1(b) is a cross-sectional view of FIGURE 
1(a); 

FIGURE 1 (c) is another embodiment of the present 
20 invention with the LCD and sensor array configured 
on the same substrate; 

FIGURE 1(d) sets forth a block diagram of the 
image display/processing apparatus arranged for 
morphological processing; 

25 FIGURES 2(a)-2(d) provide illustrations of morpho- 
logical processing operations; 
FIGURE 3 shows a sequence of operations for 
processing an image by morphological filtering, 
according to the present invention; and 

30 FIGURES 4(a) - 4(b) set forth sequences of opera- 
tions for processing images by morphological filter- 
ing according to the present invention. 

The technologies involved in two-dimensional (2-D) 

35 displays and two-dimensional (2-D) image sensors are 
quite compatible, especially when large-area amor- 
phous-or poly-silicon electronics are employed. Both 
technologies use very similar configurations and driving 
techniques. Typically, one line of these arrays are driven 

40 at a time, for LCDs the values for each element in a line 
are loaded and then the next line is addressed. Similarly, 
in the sensor array each line is addressed and a value 
of each element is read out. The combination of the 
technologies allows for powerful image capture, display, 

45 and processing. One application involving the integra- 
tion of the 2-D sensor array with the 2-D image display 
allows for a system to be constructed which can be used 
as an ordinary flat panel display, but which is also capa- 
ble of powerful morphological image processing. 

so FIGURES 1(a)-(b) provide illustrations of portions 
of an LCD and photodiode array forming the image dis- 
play/processing device A of the subject invention. In 
FIGURE 1(a) a top view of a portion of the combined 
arrays is depicted, while FIGURE 1(b) sets forth a cross- 

55 sectional view, of FIGURE 1 (a). To facilitate discussion, 
with attention to Fl GU RE 1 (b) , a person using image dis- 
play/processing device A would be viewing the device 
from the top of the page down onto the device. A light 
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flux, which acts to illuminate the LCD, is directed to de- 
vice A on the side opposite that to be viewed. 

In the embodiment shown in FIGURE 1(b), LCD B 
and photodiode array C are fabricated on two separate 
substrates 10, 12. The integrated system is comprised 
of LCD B configured with amorphous or poly-silicon ma- 
terial on top of which twodimensional sensor array C is 
aligned and bonded. The LCD B consists of a glass sub- 
strate 10 on whose first side 14 TFT circuitry 16 is fab- 
ricated. A layer of liquid crystal material 18 is placed on 
side 1 4 on top of the TFT circuitry 1 6. A thin glass cover 
plate 22 coated with a transparent conductive coating 
(e.g. I TO) 24 is placed over the liquid crystal material 
18, with ITO 24 acting as a counter electrode for LCD 
B. A thin polarizing plate 26 is bonded to the ITO coated 
cover plate 22. Alternatively, the transparent conductive 
coating may be directly deposited onto the polarizing 
plate 26. 

Appreciating that other LCD constructions may be 
possible, an important aspect of integrating the LCD B 
and sensor array C is to maintain the overall gap dis- 
tance 28 between the sensor array plane 30 and the 
LCD plane 32 small, on the order of an LCD pixel pitch, 
in order to obtain acceptable imaging. If the gap distance 
28 is too large light diffraction from a pixel increases and 
spreads out in an undesirable manner, as will be ex- 
plained in greater detail. 

In FIGURE 1 (b), two-dimensional sensor array C is 
bonded directly to polarizing plate 26. Sensor elements 
34 are bonded to LCD B such that a bottom side 36 
which has sensing capabilities is mounted to polarizing 
plate 26. Through this arrangement sensor elements 34 
sense images developed on LCD B. The pitch of the 
sensor array C is commensurate with that of LCD B, and 
as more clearly depicted in FIGURE 1 (a), the two arrays 
are aligned so that each LCD pixel 38 is to be associated 
with one corresponding sensor element 34. As previ- 
ously noted the gap distance 28 is desirably small. Par- 
ticularly, this distance is to be such that light from one 
LCD pixel 38 will impinge upon only one associated sen- 
sor element of the plurality of sensor elements 34. 

As also depicted in FIGURE 1(a), LCD TFTs 16 are 
located at intersections of horizontal lines X,, X^, X„ and 
vertical lines Yj, Y B , Y iih Y n . Sensor elements 34 are lo- 
cated at intersections of horizontal lines X j( X? n and ver- 
tical lines Y'j, Y^, Y' n . In the subject device A the arrays 
are aligned in relationship to each other such that output 
from one LCD pixel 38 projects fully onto a correspond- 
ing sensor element 34. 

It is to be appreciated that to increase the opera- 
tional speed of device A the display implemented as 
LCD B can also be configured using ferroelectric display 
technology, and the TFTs may be laser-crystallized 
TFTs. 

In another embodiment, as shown in FIGURE 1(c), 
sensor elements 34 are fabricated on the same sub- 
strate 1 0 as TFTs 1 6. Therefore, in this embodiment only 
a single substrate is needed and the bonding of the sen- 



sor array substrate discussed in connection with FIG- 
URE 1 (b) is not required. Additionally, in this embodi- 
ment sensor elements 34 are fabricated so that the side 
of the sensor elements 34 which has sensing capabili- 

s ties is facing towards the top of the page. Also, in this 
embodiment a partially reflective interface 31 may be 
included to provide an improved optical path back to 
sensor elements 34. In this manner sensor elements 34 
sense the output of LCD B. 

10 in yet another embodiment sensor elements 34 de- 
picted in FIGURE 1(b), are configured to not only have 
sensing capabilities on bottom side 36 but also on top 
side 36'. 

In the embodiment discussed in connection with 

15 FIGURE 1(c) and the embodiment immediately above 
i.e., having sensor elements 34 with sensing capabilities 
on both bottom 36 and top 36' sides, the subject inven- 
tion includes the capability of scanning documents lo- 
cated on a side opposite LCD B. The control of such a 

20 scanning operation is controlled by implementing known 
scanning control operations. 

For ease of understanding, a plurality of operational 
properties of the system are defined. It is to be appreci- 
ated that while certain characteristics of the system are 

25 set forth below, may of these properties may be relaxed 
and/or implemented in other arrangements. First, a pho- 
todiode array is implemented as the sensor array C. It 
is also possible the sensor array may be constructed 
using a photoconductor array. Also, driving of LCD B 

30 and sensor array C are synchronized, in the sense that 
whenever the sensor array C is read out, a line of sensor 
array C is read at the same time as its corresponding 
LCD line is updated. Thus, in effect, there is a constant 
time delay for each sensor element 34, between the time 

35 that one sensor element 34 corresponding one LCD pix- 
el 38 is addressed and the time that an output of the 
corresponding sensor element 34 is measured. Typical- 
ly, LCD response time is much slower than that for pho- 
todiode sensors. 

40 a further parameter is that sensor elements 34 may 
be interrogated after any N frames of integration. That 
is, sensor elements 34 may be read out each LCD frame 
update, after two frames of LCD update, etc. For the fol- 
lowing discussion transition time of the LCD B will be 

45 ignored under the assumption that LCD response time 
is less than one frame period, so that reading sensor 
elements 34 for each LCD frame produces meaningful 
data concerning the LCD output from the previous 
frame. A frame is defined as the time period it takes for 

50 the system to sequence through its operation to again 
address a previously addressed location, i.e. a pixel. It 
is a straight forward operation to extend the concepts 
described below to the case where the LCD response 
time is a significant portion of a frame period. 

55 Attention is now directed to using sensor array C of 
the image display/processing device A for parallel im- 
age processing. To use the subject image dis- 
play/processing device A for morphological image 
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processing, several attributes as shown in FIGURE 1(d) 
are included. The sensor array C is dynamically pro- 
grammable to integrate over a specified number of LCD 
frames using host computer D to control driving elec- 
tronics 40 to program the sensing elements 34 through s 
integrating signal line 50. It is to be appreciated that driv- 
ing electronics may be incorporated as part of host com- 
puter D. 

Sensor array output is fed back to LCD B for display 
and processing by feedback lines 52 which are associ- 
ated with each sensor element 34 and its corresponding 
pixel 38. This feedback arrangement has very little de- 
lay, i.e. on the order of 1 LCD frame. By this arrangement 
feedback can be accomplished without any need for an 
intermediate storage of the output from sensor array C. 
In FIGURE 1(d) only one line 52 is shown in order to 
simplify the figure, however, it is to be appreciated that 
various known feedback arrangements may be imple- 
mented. It is also to be appreciated a storage device can 
be implemented to provide an intermediate storage if 
desired. 

The levels of the sensor elements 34 of sensor ar- 
ray C are thresholded to binary, with a dynamically pro- 
grammable threshold level using thresholding line 54 
controlled by host computer D. 

Finally, driving electronics 40 are selected such that 
with minimal intervention of the computer D driving LCD 
B, the image at LCD B can be shifted by one pixel in 
either the x- or y-directions using LCD shift lines 56 and 
58. Shifting the image orthogonal to the scanline direc- 
tion of LCD B is easily accomplished, as it corresponds 
to a simple shift in the scanline sequence with respect 
to the start of a frame. Shifting the image along the scan- 
line direction will depend upon the specific manner in 
which the LCD B is driven. Using a system as described 
above, the fundamental morphological processing op- 
erations of DILATION and EROSION may be imple- 
mented. 

Prior to a specific discussion regarding such imple- 
mentation it is noted that the present discussion deals 
with binary images. 1 In this context, the term "image" re- 
fers to a representation of a two-dimensional data struc- 
ture composed of pixels. A binary image is an image 
where a given pixel is either ON or OFF. Binary images 
are manipulated according to a number of operations 
wherein one or more source images are mapped onto 
a destination image. The results of such operations are 
generally referred to as images. The image that is the 
starting point for processing is sometimes referred to as 
the original image. 

Pixels are defined to be ON if they are black and 
OFF if they are white. In some cases a "donl care" pixel 
can occur which may be either an ON or an OFF pixel. 

A "solid region" of an image refers to a region ex- 
tending many pixels in both dimensions within which 
substantially all the pixels are ON. 

A "textured region" of an image refers to a region 
that contains a relatively finegrained pattern. Examples 



of textured regions are halftone or stippled regions. 

AND, OR, and XOR are logical operations carried 
out between two images on a pixel-by-pixel basis. 

NOT is a logical operation carried out on a single 
image on a pixel-by-pixel basis. 

Prior to considering the operation of the image 
processing in further detail, it may be helpful to briefly 
define some of the terms that have been adopted for 
describing "morphological image processing opera- 
tions": 

"Morphological operation" is an operation on a bit- 
map image (source image) that uses a local rule at each 
pixel location with the source image to create another 
bitmap image (destination image). For convenience, the 
source and destination images sometimes are referred 
to as "pixel -map" images so that the operational rate can 
be viewed as operating on each "pixel". "Bitmap" and 
"pixelmap" are synonymous terms for a data structure 
of a certain type, and "bit" and "pixel" are used inter- 
changeably to describe the contents of such a data 
structure. 

■Structuring Element" (SE) is an image object, typ- 
ically of relatively small size and simple shape, for prob- 
ing the source image to extract information from it 
through the use of selected morphological operations. 
The SEs referred to herein are binary SEs. They are il- 
lustrated by using solid areas to identify their ON pixels 
and hollow areas to identify their OFF pixels. Their cent- 
er locations are identified by a ("X"). 

A number of morphological operations map a 
source image onto an equally sized destination image 
according to a rule defined by a structuring element 
(SE). The SE is defined by the center location ("X") and 
a number of pixel locations, each having a defined value 
(ON or OFF). The pixels defining the SE do not have to 
be adjacent each other. The center location ("X") need 
not be at the geometrical center of the pattern and does 
not even need not even be inside the pattern. 

A "solid" SE refers to an SE having a periphery with- 
in which all pixels are ON. For example, a solid 2x2 SE 
is a 2x2 square of ON pixels. A solid SE need not be 
rectangular. 

The following terms are specific to binary morpho- 
logical operations: 

"EROSION" is an operation that is performed by 
probing a binary source image with a SE to write an ON 
(1) or an OFF (0) pixel into the destination image for 
each pixel location within the source image, with the log- 
ic level of the pixel that is written at any given location 
depending upon whether the SE is matched or not by 
the source image when it is centered on the given pixel 
location. When the SE to be matched contains both 
"hits" and "misses," the matching operation commonly 
is called "hit-miss transform." However, to simplify this 
disclosure, the definition of EROSION has been ex- 
panded to include such hit-miss transforms. 

"DILATION" is an operation that is performed by 
probing a binary source image with a SE to write the SE 
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into the destination image on centers corresponding to 
the location of all 'ON" pixels on the source image. As 
used herein, DILATION is defined only for "hits" in the 
SE, so "misses" are ignored. Thus, the dilated destina- 
tion image is the union of all replicas of the SE translated 
to all 1 -pixels of the source image. 

"OPENING" is an operation for replicating a SE in 
the destination image for each match to the SE in the 
source image. It is equivalent to an EROSION of a 
source image by a SE followed by a DILATION of the 
eroded image by the same SE. In keeping with the fore- 
going definitions of EROSION and DILATION, the defi- 
nition of the OPENING operation has been expanded to 
include an EROSION with a SE containing both "hits" 
and "misses" followed by a DILATION with only the "hits" 
in the SE. 

"CLOSING" is an operation composed of a DILA- 
TION of a source image followed by an EROSION of the 
dilated image. A CLOSING of an image is equivalent to 
a bit inversion of an OPENING that is performed on a 
bit inverted source image. In view of the foregoing def- 
inition of DILATION, it will be understood that a CLOS- 
ING is defined herein only for "hits" in the SE, so any 
"misses" are ignored. 

Morphological operations are translationally invari- 
ant. In other words, a source image may be translated 
prior to being transformed, thereby causing the result to 
be translated or shifted by the same amount, without 
otherwise changing it. This means that these operations 
may be implemented with a high degree of parallelism 
because each bit or pixel in the source image is proc- 
essed in accordance with the same rule. 

EROSION, DILATION, OPENING and CLOSING 
operations performed with SE's consisting of only "hits" 
are geometrically "increasing" operations. Therefore, if 
a first image is contained in a second image, any of 
these operations that are performed with such a SE on 
the first image will also be contained in the second im- 
age. Furthermore, CLOSING is "extensive", and OPEN- 
ING is "antiextensive". Accordingly, the source image is 
contained in the destination image when the source is 
transformed by a CLOSING, and the destination image 
is contained in the source image when the source image 
is transformed by an OPENING. The results of OPEN- 
ING and CLOSING operations are independent of the 
position of the center of the SE. Moreover, OPENING 
and CLOSING operations are independent, which 
means they will not change the transformed image if 
they are reapplied to it. Other terms that are sometimes 
used in describing morphological operations are: 

a "4-connected region" is a set of ON (1) pixels, 
such that a path between any two of those pixels 
can be found that stays entirely within the set of ON 
pixels and consists of only horizontal or vertical 
1 -pixel moves, 

a "8-connected region" is a set of ON (1) pixels, 
such that a path between any two of those pixels 



can be found that stays entirely within the set of ON 
pixels and consists of only horizontal, vertical or 
diagonal 1 -pixel moves. 

5 A "hit-miss" SE is a SE that specifies a non-zero set 
of ON pixels and OFF (0) pixels, with those two sets be- 
ing nonoverlapping (i.e., non-intersecting). A "weakly" 
matched filter specifies relatively few pixels of the pixel 
pattern to which it is matched, while a "strongly" 
matched filter specifies a large percentage of the pixel 
pattern to which it is matched. 

A "hit-only" SE is an SE that specifies a non-zero 
set of ON pixels. 

"Fillclip" is a morphological operation where one im- 
age is used as a seed and is "grown" morphologically, 
clipping it at each growth step to the second image. 

"Fill8" is an image operation in which an 8-connect- 
ed region is filled to minimum rectangular bounding box- 
es. 

A "mask" refers to an image, normally derived from 
an original image, that contains substantially solid re- 
gions of ON pixels corresponding to regions of interest 
in the original image. The mask may also contain re- 
gions of ON pixels that do not correspond to regions of 
interest. 

The various operations defined above are some- 
times referred to in noun, adjective, and verb forms. For 
example, references to DILATION (noun form) may be 
in terms of DILATING the image or the image being DI- 
LATED (verb forms) or the image being subjected to a 
DILATION operation (adjective form). No difference in 
meaning is intended. 

The most straightforward approach to binary mor- 
phological processing is to define structuring element 
(SE) 42 of FIGURE 2(a) having center location 44. Orig- 
inal image 46 of FIGURE 2(b) may be ERODED or DI- 
LATED according to structuring element (SE) 42. DILA- 
TION corresponds to multiple shifts of the original image 
46 as prescribed by structuring element (SE) 42, fol- 
lowed by a pixelwise logical OR operation over multiple 
images. EROSION is the result of multiple inverted 
shifts associated with structuring element (SE) 42, fol- 
lowed by a logical AND operation. Original image 46 of 
Fig. 2(b) is shifted according to the location of the black 
pixels in structuring element (SE) 42, with respect to 
center location pixel 44. 

In this case, for DILATION three images are creat- 
ed: the original 46, a copy shifted by an amount of (Ax, 
Ay) = (1 , 1 ) and a copy shifted by (Ax, Ay) = (-1 , 1 ). Logical 
so ORing of the image produces the DILATION result 
shown in Fig. 2(c). For EROSION the original image un- 
dergoes a logical AND operation with image copies by 
(Ax, Ay) = (-1.-1) and (Ax, Ay) = (1,-1). The results of 
such processing being shown by FIGURE 2(d). 
55 There are a variety of ways to implement morpho- 
logical processing with image display/processing device 
A of the subject invention. 

Assuming a given structuring element S(i,j), the 
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structuring element (SE) 42 is described by a series of 
N translations of the original image 46. These N trans- 
lations are denoted by T k , where k runs from 1 to N. For 
example, the structuring element (SE) 42 of FIGURE 2 
(a) could be described by translation = (1,1), T 2 = (- 
1,1), with N = 2. Inverted shifts are denoted by T. k , i.e. 
T_! = (-1,1), T_ 2 = (1,-1) in FIGURE 2(a). Given the 
translations associated with structuring element (SE) 
42, a DILATION may be performed as described below. 
The original image (assuming it is binary) is loaded to 
the LCD B. At the time of loading, the charge of sensor 
elements 34 is zeroed. After a time AT (ideally equal to 
a single frame time of the LCD), the image translated by 
T-, is loaded. This is done by calling out the desired shift 
in x and y, which is performed in real time by the LCD 
driving electronics 40 through shift lines 56, 58. After 
another time AT, the levels of sensing elements 34 of 
sensor array C are read. Sensor signals will then corre- 
spond to the sum of original image and the image shifted 
by T v This sensor data is thresholded using threshold- 
ing line 54 so that a level corresponding to a single pixel 
turned ON in LCD B for a time AT yields a 1 . This image 
is sent to LCD B, while again zeroing the sensor ele- 
ments 34 of sensor array C. After a next time AT of this 
image, LCD B is loaded with the original image 46, shift- 
ed by T 2 . After time AT of the T 2 image, the sensor levels 
or data is read out, thresholding in the same manner as 
above. The result will be the DILATED image of FIGURE 
2(c). 

I n the following discussion an "ON" pixel is intended 
to mean it is in a darkened state. As described in con- 
junction with FIGURE 3, for a series of shifts T k , the fol- 
lowing steps are performed: 

1. Load LCD with original image, 100. Zero sensor 
elements, 102. Integrate image at sensor elements 
for time AT (a predetermined LCD frames), 104. 

2. Load LCD with image shifted by T 1 (where , is 
a predetermined time period), 106, integrate image 
(T-,) at sensor elements for time AT 108, then read 
out sensor elements, 110. Threshold sensor data 
so that a single LCD pixel turned ON for period AT 
or greater will produce a (1), 112. 

3. Load LCD with sensor image, 114. Zero sensor 
elements, 116. Integrate image at sensor elements 
for time AT, 118. 

4. Load LCD with original image shifted by T 2 
(where T 2 is a second predetermined time period), 
120. Integrate image at sensor elements for time 
AT, 122, then read out sensors, 124. Threshold sen- 
sor data so that a single LCD pixel turned ON for 
period AT or greater will produce a (1), 126. 

5. Repeat steps 3 and 4, for all T k of the structuring 
element (SE), 128. When finished, the final sensor 
image to host computer, 130. 

For EROSION, a comparable set of operations is 
employed, with shifts T k replaced by shifts T. k , and with 



the thresholding changed so that a value of 1 is output 
by the sensor elements only if the signal corresponds to 
an LCD pixel turned ON, for a period of 2AT. 

In the above steps, the sensor elements integrate 
5 over pairs of LCD images, and their thresholds are de- 
fined independently of the structuring element (SE) 
(specifically, independent of the number of translations 
N associated with the structuring element). An alterna- 
tive approach as illustrated in FIGURE 4(a), is to inte- 
10 grate the sensor signal over all translations T k of the 
LCD image. In this case, for DILATION, the sensor ele- 
ments are zeroed, 1 34, and then LCD images are load- 
ed sequentially, at a rate of 1/AT, 136, while the sensor 
elements integrate the images, 138. After time (N + 1) 
15 AT, the sensor array is read out, 140, and thresholded 
to a value of (1 ) if the sensor level corresponds to a pixel 
being in the ON state of the LCD for a period AT or great- 
er, with other signals thresholded to zero, 142. 

For EROSION, as depicted in FIGURE 4(b), the set 
20 of N images translated by T. k is displayed on the LCD, 
at a rate of 1/AT, 144. After time (N + 1)AT, the sensor 
array is read out, 146, and thresholded to a value of (1 ) 
if the sensor level corresponds to a pixel being in the 
ON state of the LCD for a period (N + 1 )AT, with all other 
25 signals thresholded to zero, 148. Using this approach 
there is no need to load the LCD with a sensor image. 

With the capability of performing EROSIONS and 
DILATIONS, and therefore OPENING and CLOSING 
operations, many important processing operations may 
30 be performed. In addition, image reduction may be ac- 
complished, for example, by sampling the LCD image 
at every 2 n scan of the pixels, and directing this image 
back to LCD B. This is often a very useful processing 
operation. Finally, the system subject can be used for 
35 greyscale morphology and processing. The combined 
display/processing device A could thus be used to per- 
form a wide range of image processing functions with 
high speed. Many of these functions would be associ- 
ated with document recognition, manipulation, etc. The 
40 system can function as a normal display as well. 



Claims 

45 1. An image display/processing apparatus 
comprising : 

a liquid crystal display (LCD) (B) having an 
array of switching elements (16), correspond- 

50 ing to a plurality of LCD pixels; 

a sensor array (C) integrated with the LCD to 
form a single unit, the sensor array having a plu- 
rality of sensor elements (34) positioned in rows 
and columns in the array, the array of switching 

55 elements and the array of sensor elements 

aligned with each other to allow a given LCD 
pixel to project onto a corresponding sensor 
element of the array of sensor elements; and 
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driving circuitry (40) in operative connection 
with both the LCD and sensor array. 

2. The apparatus according to claim 1 further including 
morphological image processing means (D) to con- s 
trol operation of the integrated LCD and sensor 
array arrangement. 

3. The image display/processing apparatus of claim 2, 
wherein: 10 



4. The apparatus according to claim 1 , 2 or 3 wherein 
the LCD and sensor array are formed (1) on two 20 
separate substrates (10,12) and are then inte- 
grated, or (2) on a single substrate (10). 

5. The apparatus according to any of the preceding 
claims wherein a gap (28) between the sensor array 25 
and LCD is equal to or less than on the order of an 
LCD pixel pitch. 

6. The apparatus according to any of the preceding 
claims wherein the sensor elements (34) include 30 
sensing capabilities on a first side (36) and a second 
side (36'), whereby scanning of a document placed 

on the second side of the sensor elements is 
achieved. 

35 

7. The apparatus according to any of the preceding 
claims wherein the sensor array further includes a 
means for dynamically programming the sensor 
array to integrate a sensing current. 

40 

8. The apparatus according to any of the preceding 
claims wherein the single unit further includes: 



whereby the first side of the sensing elements 
is directed towards the LCD. 

9. The apparatus according to any of the preceding 
claims further including a transferring means for 
transferring an output from the sensor array directly 
to the LCD, thereby eliminating intermediate stor- 
age of sensor element output. 

10. A method for displaying and processing an image 
on an integrated display/processing apparatus 
comprising the steps. 

loading a liquid crystal device (LCD) (B) with 
image data, the LCD being in the form of an 
array with switching elements (16) positioned 
at rows and columns and with a pixel associ- 
ated with each of the switching elements; 
displaying an image by adjusting each of the 
pixels in accordance with the loaded image 
data; 

sensing data of the displayed image by a sen- 
sor array (C) comprised of a plurality of sensing 
elements (34) located at rows and columns, the 
sensor array integrated as a single unit with the 
LCD, the sensor array and LCD aligned with 
each other to allow a given LCD pixel to project 
fully onto one sensing element of the sensing 
array; 

feeding back data from the sensor array to the 
LCD based on the data of the displayed image; 
and 

processing the feedback data of the displayed 

image. 



the switching elements (16) are thin film tran- 
sistors (TFTs); 

the sensor elements (34) are photodiodes; the 
apparatus further comprising: 15 
a feedback circuit (52) to feed back to the LCD 
data obtained by the photodiodes. 



a glass substrate (10) having the switching ele- 
ments (16) fabricated thereon as thin film tran- *s 
sistors (TFTs); 

a layer (18) of liquid crystal material applied on 
top of the TFTs; 

a glass coverplate (22) coated with a transpar- 
ent coating (24) placed over the liquid crystal so 
material; 

a thin polarizing plate (1 2) bonded on a first side 
to the transparent conductive coating; and 
said sensor elements (34) having a first side 
(36) and a second side (36') with at least the ss 
first side having sensing capabilities, and the 
sensing elements bonded, by the first side, to 
a second side of the thin polarizing plate, 
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